Odysseus/DFS: Integration of DBMS and Distributed File System for Transaction Processing of Big Data

摘要

The relational DBMS (RDBMS) has been widely used since it supports varioushigh-level functionalities such as SQL, schemas, indexes, and transactions thatdo not exist in the O/S file system. But, a recent advent of big datatechnology facilitates development of new systems that sacrifice the DBMSfunctionality in order to efficiently manage large-scale data. Those so-calledNoSQL systems use a distributed file system, which support scalability andreliability. They support scalability of the system by storing data into alarge number of low-cost commodity hardware and support reliability by storingthe data in replica. However, they have a drawback that they do not adequatelysupport high-level DBMS functionality. In this paper, we propose anarchitecture of a DBMS that uses the DFS as storage. With this novelarchitecture, the DBMS is capable of supporting scalability and reliability ofthe DFS as well as high-level functionality of DBMS. Thus, a DBMS can utilize avirtually unlimited storage space provided by the DFS, rendering it to besuitable for big data analytics. As part of the architecture of the DBMS, wepropose the notion of the meta DFS file, which allows the DBMS to use the DFSas the storage, and an efficient transaction management method includingrecovery and concurrency control. We implement this architecture inOdysseus/DFS, an integration of the Odysseus relational DBMS, that has beenbeing developed at KAIST for over 24 years, with the DFS. Our experiments ontransaction processing show that, due to the high-level functionality ofOdysseus/DFS, it outperforms Hbase, which is a representative open-source NoSQLsystem. We also show that, compared with an RDBMS with local storage, theperformance of Odysseus/DFS is comparable or marginally degraded, showing thatthe overhead of Odysseus/DFS for supporting scalability by using the DFS as thestorage is not significant.